A SOFTWARE ENGINE TO JUSTIFY AN EXPERT SYSTEM’S CONCLUSIONS FOR DETECTING RENAL OBSTRUCTION FROM Tc-99m MAG3 SCANS

Objectives: We have previously developed a renal expert system (RENEX) as a decision support tool to assist in the interpretation of Tc-99m MAG3 scans to detect renal obstruction in patients referred for diuresis renography. A decision support system may give the right answer for the wrong reasons. This study is to describe and evaluate a software engine to justify the conclusions reached by RENEX for assessing patients with suspected renal obstruction and to learn new knowledge from this evaluation that can be incorporated back into RENEX for improved diagnostic performance. Methods: RENEX consists of sixty heuristic rules (IF A THEN B) extracted from the domain expert to generate the knowledge base and a forward chaining inference engine in order to determine obstruction. The justification engine keeps track of the sequence of each rule that is instantiated to reach a conclusion. Once the final conclusions are reached, they are reported in a concatenated set of sentences stringing the conclusions together; the key conclusions are underlined. The interpreter can then request justification for a specific conclusion by clicking on the underlined word. The justification process then reports the English translation of all concatenated rules instantiated to reach that conclusion. The justification engine was evaluated using a prospective group of 60 patients (117 kidneys). After reviewing the standard renal pre and post furosemide MAG3 scans together with associated quantitative parameters generated by QuantEM 2.0, a blinded expert determined if each kidney was obstructed, equivocal or not obstructed and identified and ranked the main variables associated with each interpretation. Two parameters were then tabulated: 1) the frequency the main variables associated with obstruction by the expert were also justified by RENEX and 2) the frequency that the justification rules provided by RENEX were deemed to be correct by the domain expert. Only when RENEX and the domain expert agreed on the diagnosis (87 kidneys), were the results used to test the justification. Kidneys interpreted as equivocal (22) or disagreements between RENEX and the domain expert were analyzed on a case-by-case basis to generate new knowledge. Results: In these 87 kidneys where there was agreement, RENEX agreed with 91% (184/203) of the rules supplied by the expert for justifying the diagnosis. RENEX provided 103 additional rules justifying the diagnosis: the expert agreed that 102 (99%) were correct although the rules were considered of secondary importance. Conclusions: We have described and evaluated a software engine to justify a renal expert system’s conclusions for detecting renal obstruction using pre and post furosemide Tc99m MAG3 renal scans. This tool is expected to increase physician confidence in the interpretations provided by RENEX, assist physicians and trainees gain a higher level of expertise and, importantly, aid the developers improve the diagnostic performance of the decision support system. INTRODUCTION The time a physician has to devote to individual clinical studies continues to shrink due to an increased procedure volume and related paper-work as well as the time required assimilating an ever-expanding knowledge base. Moreover, t the information that needs to be assimilated to interpret each study continues to increase due to the clinical data and past studies available in expanded digital storage of the patient’s records as well as the increased number of images that are acquired and have to be reviewed due to improved temporal and spatial resolution. Decision support systems have been suggested as an artificial intelligence tool to help physicians interpret diagnostic studies at a high level of expertise while limiting the time needed for the interpretation. We have previously developed a renal expert system (RENEX) as a decision support tool to assist in the interpretation of Tc-99m MAG3 scans to detect renal obstruction in patients referred for diuresis renography (1). We chose to develop a decision support system to detect renal obstruction from Tc-99m-MAG3 renography for two reasons: (1) the vast majority of the 590,000 renal scans performed annually in the United States are performed with Tc-99m MAG3 and (2) many are interpreted by diagnosticians in sites that perform less than three studies per week (2). The exposure to a limited number of diuresis renography studies makes it difficult for physicians to develop the needed diagnostic expertise. Expert systems have been investigated in nuclear medicine to assist in the interpretation of perfusion-ventilation lung studies (3) and HMPAO brain SPECT studies (4). We have also developed (5) and extensively validated (6) an expert system called PERFEX (for perfusion expert) as a tool for the computer-assisted diagnosis of stress/rest myocardial perfusion SPECT. One of the main reasons we have chosen the expert system approach is that it allows for justification of the conclusions reached by the system. This justification process is important because it provides more experienced diagnosticians with the rationale for the diagnosis, allowing them to agree or disagree with the conclusions based on whether or not they agree or disagree with the justification (reasons) for the conclusions. For the less experienced (less knowledgeable) diagnosticians, the justification process provides an opportunity for training on a case-by-case basis by teaching them the specific established rules that applied in each case. A decision support system may give the right answer for the wrong reasons. The benefits of the justification process described above are only true if the expert system reached the correct diagnosis through the correct reasoning process. This study is to describe and evaluate a software engine to justify the conclusions reached by RENEX for assessing patients with suspected renal obstruction and to learn new knowledge from this evaluation that can be incorporated back into RENEX for improved diagnostic performance. METHODS The justification engine is a software module of the RENEX expert system. Thus, the acquisition protocol and data analysis methods described below are the same as those used by RENEX (1). The patient population used for this validation is different than the training group used to develop RENEX and as such represents a prospective population. Patients Renal studies from 60 patients (28 males, 32 females, mean age = 53.8 + 17.6, 117 kidneys) were used as a pilot group to test the RENEX justification engine. All studies used for this development were obtained from the renal database of patients referred to our nuclear medicine service to evaluate suspected renal obstruction. This study was performed under the purview and approval of Emory’s Internal Review Board. Patients were selected because their studies included a baseline Tc-99m MAG3 dynamic study followed by a furosemide challenge. Acquisition Protocol Patients were positioned supine, with the scintillation camera detector placed under the table. A three-phase dynamic acquisition (baseline scan) was begun as approximately 10 mCi of Tc-99m MAG3 were injected; phase one consisted of 24 2second frames, phase two was 16 15-second frames, and phase three was 40 30-second frames. For 44 of the patients in the study, review of the baseline scan could not exclude obstruction and these 44 patients received an intravenous injection of approximately 40 mg of furosemide followed immediately by a second single-phase 20 min dynamic acquisition consisting of 40 30-second frames. For the other 16 patients, the baseline study excluded obstruction and they did not receive furosemide. Data Analysis All patient studies were processed using the QuantEMTM 2.0 software, an improved version of the renal quantification program (7). The QuantEM software, developed specifically for Tc-99m MAG3, incorporates several quality control procedures to improve reproducibility, generates specific quantitative parameters recommended for scan interpretation and allows the MAG3 clearance to be calculated using a camera based technique. QuantEMTM has been previously extensively validated in a multicenter trial (8). For the baseline renogram, a static image is summed from the 2-3 minute postinjection frames. Using a filtered version of this image, whole kidney, background and cortical regions of interest are automatically defined. The user can override any of these automatic ROIs and replace them with manual ROIs. Background-subtracted curves are generated for the whole kidney and 47 quantitative parameters are generated including patient demographics (height, weight, age, sex, body surface area), curve parameters (time to peak counts, and 20 min to maximum count ratio for both whole kidney and cortical ROIs), voiding indices (post void to pre void and post void to maximum count ratios) and the MAG3 clearance. The MAG3 clearance is calculated from the 1-2.5 minute whole kidney MAG3 counts, and the pre-injection and post-injection images of the dose syringe. For the diuretic study, a static image is summed from the 1-5 minute postinjection frames. Regions of interest are manually drawn for the whole kidney, background and renal collecting system. Background-subtracted curves are generated for the whole kidney and renal pelvis, and times-to-half-peak are calculated. After processing the diuretic study, the baseline renogram results are loaded and RENEX calculates ratios comparing the first-minute and prevoid (last minute) counts in the diuretic acquisition to the 1-2 minute counts and peak counts in the baseline acquisition. The RENEX Expert System RENEX has been previously described elsewhere (1). Briefly, normal limits were established for 47 quantitative parameters extracted from the Tc-99m MAG3 scans of 100 potential renal donors (9). From these data the domain expert estimated 5 boundary conditions for each parameter: (1) definitely abnormal, (2) probably abnormal, (3) equivocal, (4) probably normal and (5) definitely normal. A sigmoid-type fit was then performed constrained to these 5 boundary conditions creating a parameter knowledge library used for converting the value of prospective patient’s individual quantitative parameters to a certainty factor (CF). The CFs indicate the degree of certainty that each parameter value is abnormal or normal and, therefore, consistent or inconsistent with disease. The CF values range between -1 (definitely normal) to +1 (definitely abnormal) with the interval between -.2 and +.2 representing missing, equivocal or indeterminate parameters. Sixty heuristic rules (IF A THEN B) were extracted from the domain expert to generate the knowledge base for detecting obstruction. A forward chaining inference engine was developed to determine obstruction. The inference engine is a computer algorithm that uses specific equations known as the MYCIN combinatories (an approximation of Bayes theorem) in order to combine the certainty that a parameter (or parameters) is abnormal with the certainty of a rule to modify the certainty that a hypothesis is true (a parameter is abnormal or a kidney is abnormal, i.e., obstructed) (10).